The present invention relates to automated system testing, and more specifically, to automated system testing in a complex software environment.
In a product development environment, large numbers of automated tests are executed and results are checked against expected results in order to ensure new code delivered for the product has not caused any regressions or injected new problems. These tests are run repeatedly for every build of the product code.
The test framework for these tests is often very complex, where the test itself involves a complex scenario such as setting up the test artifacts and going through multiple steps of transaction processing in systems. For example, the test framework may start one or multiple product systems, define the resources in the systems, look up a user identifier and password from the framework and then use this to log on to the systems and then start a transaction that starts subsequent processing within the product system(s).
When such a test fails, it is often very time consuming and difficult to work out where the failure is and what the cause is. The output from the systems under test, e.g. messages, trace, dump etc., has to be examined by product experts and test experts in order to work out whether the problem is with the system(s) under test or with the test framework itself. This can be a slow process and can result in significant delays in executing further tests with a concomitant impact on the ability of development organization to deliver further code.